
<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

    <title>第四章 分组 &#8212; Joyful Pandas 1.0 documentation</title>
<script>
  document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
  document.documentElement.dataset.theme = localStorage.getItem("theme") || "light"
</script>

  <!-- Loaded before other Sphinx assets -->
  <link href="../_static/styles/theme.css?digest=92025949c220c2e29695" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=92025949c220c2e29695" rel="stylesheet">


  <link rel="stylesheet"
    href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
  <link rel="preload" as="font" type="font/woff2" crossorigin
    href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
  <link rel="preload" as="font" type="font/woff2" crossorigin
    href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">

    <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="../_static/plot_directive.css" />
    <link rel="stylesheet" type="text/css" href="../_static/css/s4defs-roles.css" />

  <!-- Pre-loaded scripts that we'll load fully later -->
  <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=92025949c220c2e29695">

    <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
    <script src="../_static/jquery.js"></script>
    <script src="../_static/underscore.js"></script>
    <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="../_static/doctools.js"></script>
    <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="第五章 变形" href="ch5.html" />
    <link rel="prev" title="第三章 索引" href="ch3.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
  </head>
  
  
  <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="180" data-default-mode="">
    <div class="bd-header-announcement container-fluid" id="banner">
      

    </div>

    
    <nav class="bd-header navbar navbar-light navbar-expand-lg bg-light fixed-top bd-navbar" id="navbar-main"><div class="bd-header__inner container-xl">

  <div id="navbar-start">
    
    
  


<a class="navbar-brand logo" href="../index.html">
  
  
  
  
    <img src="../_static/finallogo1.svg" class="logo__image only-light" alt="Logo image">
    <img src="../_static/finallogo1.svg" class="logo__image only-dark" alt="Logo image">
  
  
</a>
    
  </div>

  <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbar-collapsible" aria-controls="navbar-collapsible" aria-expanded="false" aria-label="Toggle navigation">
    <span class="fas fa-bars"></span>
  </button>

  
  <div id="navbar-collapsible" class="col-lg-9 collapse navbar-collapse">
    <div id="navbar-center" class="mr-auto">
      
      <div class="navbar-center-item">
        <ul id="navbar-main-elements" class="navbar-nav">
    <li class="toctree-l1 nav-item">
 <a class="reference internal nav-link" href="../Home.html">
  Home
 </a>
</li>

<li class="toctree-l1 current active nav-item">
 <a class="reference internal nav-link" href="index.html">
  Content
 </a>
</li>

<li class="toctree-l1 nav-item">
 <a class="reference internal nav-link" href="../Author.html">
  Author
 </a>
</li>

<li class="toctree-l1 nav-item">
 <a class="reference internal nav-link" href="../Datawhale.html">
  Datawhale
 </a>
</li>

<li class="toctree-l1 nav-item">
 <a class="reference internal nav-link" href="../pandas%E6%95%B0%E6%8D%AE%E5%A4%84%E7%90%86%E4%B8%8E%E5%88%86%E6%9E%90.html">
  pandas数据处理与分析
 </a>
</li>

<li class="toctree-l1 nav-item">
 <a class="reference internal nav-link" href="../%E8%A1%A5%E5%85%85%E4%B9%A0%E9%A2%98.html">
  补充习题
 </a>
</li>

    
    <li class="nav-item">
        <a class="nav-link nav-external" href="https://pandas.pydata.org/docs/index.html">Doc<i class="fas fa-external-link-alt"></i></a>
    </li>
    
</ul>
      </div>
      
    </div>

    <div id="navbar-end">
      
      <div class="navbar-end-item">
        <span id="theme-switch" class="btn btn-sm btn-outline-primary navbar-btn rounded-circle">
    <a class="theme-switch" data-mode="light"><i class="fas fa-sun"></i></a>
    <a class="theme-switch" data-mode="dark"><i class="far fa-moon"></i></a>
    <a class="theme-switch" data-mode="auto"><i class="fas fa-adjust"></i></a>
</span>
      </div>
      
      <div class="navbar-end-item">
        <ul id="navbar-icon-links" class="navbar-nav" aria-label="Icon Links">
        <li class="nav-item">
          <a class="nav-link" href="https://github.com/datawhalechina/joyful-pandas" rel="noopener" target="_blank" title="GitHub"><span><i class="fab fa-github-square"></i></span>
            <label class="sr-only">GitHub</label></a>
        </li>
      </ul>
      </div>
      
    </div>
  </div>
</div>
    </nav>
    

    <div class="bd-container container-xl">
      <div class="bd-container__inner row">
          

<!-- Only show if we have sidebars configured, else just a small margin  -->
<div class="bd-sidebar-primary col-12 col-md-3 bd-sidebar">
  <div class="sidebar-start-items"><form class="bd-search d-flex align-items-center" action="../search.html" method="get">
  <i class="icon fas fa-search"></i>
  <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
  <div class="bd-toc-item active">
    <ul class="current nav bd-sidenav">
 <li class="toctree-l1">
  <a class="reference internal" href="ch1.html">
   第一章 预备知识
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch2.html">
   第二章 pandas基础
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch3.html">
   第三章 索引
  </a>
 </li>
 <li class="toctree-l1 current active">
  <a class="current reference internal" href="#">
   第四章 分组
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch5.html">
   第五章 变形
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch6.html">
   第六章 连接
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch7.html">
   第七章 缺失数据
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch8.html">
   第八章 文本数据
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch9.html">
   第九章 分类数据
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="ch10.html">
   第十章 时序数据
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="%E5%8F%82%E8%80%83%E7%AD%94%E6%A1%88.html">
   参考答案
  </a>
 </li>
</ul>

  </div>
</nav>
  </div>
  <div class="sidebar-end-items">
  </div>
</div>


          


<div class="bd-sidebar-secondary d-none d-xl-block col-xl-2 bd-toc">
  
    
    <div class="toc-item">
      
<div class="tocsection onthispage mt-5 pt-1 pb-3">
    <i class="fas fa-list"></i> On this page
</div>

<nav id="bd-toc-nav">
    <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id2">
   一、分组模式及其对象
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id3">
     1. 分组的一般模式
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id4">
     2. 分组依据的本质
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#groupby">
     3. Groupby对象
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id5">
     4. 分组的三大操作
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id6">
   二、聚合函数
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id7">
     1. 内置聚合函数
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#agg">
     2. agg方法
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id8">
   三、变换和过滤
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#transform">
     1. 变换函数与transform方法
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id9">
     2. 组索引与过滤
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id10">
   四、跨列分组
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#apply">
     1. apply的引入
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#id11">
     2. apply的使用
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id12">
   五、练习
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#ex1">
     Ex1：汽车数据集
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#ex2-transform">
     Ex2：实现transform函数
    </a>
   </li>
  </ul>
 </li>
</ul>

</nav>
    </div>
    
    <div class="toc-item">
      
    </div>
    
  
</div>


          
          
          <div class="bd-content col-12 col-md-9 col-xl-7">
              
              <article class="bd-article" role="main">
                
  <section id="id1">
<h1>第四章 分组<a class="headerlink" href="#id1" title="Permalink to this heading">#</a></h1>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [1]: </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>

<span class="gp">In [2]: </span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
</pre></div>
</div>
<section id="id2">
<h2>一、分组模式及其对象<a class="headerlink" href="#id2" title="Permalink to this heading">#</a></h2>
<section id="id3">
<h3>1. 分组的一般模式<a class="headerlink" href="#id3" title="Permalink to this heading">#</a></h3>
<p>分组操作在日常生活中使用极其广泛，例如：</p>
<ul class="simple">
<li><p>依据 <span class="red">性别</span> 分组，统计全国人口 <span class="green">寿命</span> 的 <span class="blue">平均值</span></p></li>
<li><p>依据 <span class="red">季节</span> 分组，对每一个季节的 <span class="green">温度</span> 进行 <span class="blue">组内标准化</span></p></li>
<li><p>依据 <span class="red">班级</span> 分组，筛选出组内 <span class="green">数学分数</span> 的 <span class="blue">平均值超过80分的班级</span></p></li>
</ul>
<p>从上述的几个例子中不难看出，想要实现分组操作，必须明确三个要素：<span class="red">分组依据</span> 、 <span class="green">数据来源</span> 、 <span class="blue">操作及其返回结果</span> 。同时从充分性的角度来说，如果明确了这三方面，就能确定一个分组操作，从而分组代码的一般模式即：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="n">分组依据</span><span class="p">)[</span><span class="n">数据来源</span><span class="p">]</span><span class="o">.</span><span class="n">使用操作</span>
</pre></div>
</div>
<p>例如第一个例子中的代码就应该如下：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s1">&#39;Gender&#39;</span><span class="p">)[</span><span class="s1">&#39;Longevity&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
</pre></div>
</div>
<p>现在返回到学生体测的数据集上，如果想要按照性别统计身高中位数，就可以如下写出：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [3]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s1">&#39;data/learn_pandas.csv&#39;</span><span class="p">)</span>

<span class="gp">In [4]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s1">&#39;Gender&#39;</span><span class="p">)[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">median</span><span class="p">()</span>
<span class="gh">Out[4]: </span>
<span class="go">Gender</span>
<span class="go">Female    159.6</span>
<span class="go">Male      173.4</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
</section>
<section id="id4">
<h3>2. 分组依据的本质<a class="headerlink" href="#id4" title="Permalink to this heading">#</a></h3>
<p>前面提到的若干例子都是以单一维度进行分组的，比如根据性别，如果现在需要根据多个维度进行分组，该如何做？事实上，只需在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 中传入相应列名构成的列表即可。例如，现希望根据学校和性别进行分组，统计身高的均值就可以如下写出：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [5]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">&#39;School&#39;</span><span class="p">,</span> <span class="s1">&#39;Gender&#39;</span><span class="p">])[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gh">Out[5]: </span>
<span class="go">School                         Gender</span>
<span class="go">Fudan University               Female    158.776923</span>
<span class="go">                               Male      174.212500</span>
<span class="go">Peking University              Female    158.666667</span>
<span class="go">                               Male      172.030000</span>
<span class="go">Shanghai Jiao Tong University  Female    159.122500</span>
<span class="go">                               Male      176.760000</span>
<span class="go">Tsinghua University            Female    159.753333</span>
<span class="go">                               Male      171.638889</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
<p>目前为止， <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 的分组依据都是直接可以从列中按照名字获取的，那如果希望通过一定的复杂逻辑来分组，例如根据学生体重是否超过总体均值来分组，同样还是计算身高的均值。</p>
<p>首先应该先写出分组条件：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [6]: </span><span class="n">condition</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">Weight</span> <span class="o">&gt;</span> <span class="n">df</span><span class="o">.</span><span class="n">Weight</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
</pre></div>
</div>
<p>然后将其传入 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 中：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [7]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="n">condition</span><span class="p">)[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gh">Out[7]: </span>
<span class="go">Weight</span>
<span class="go">False    159.034646</span>
<span class="go">True     172.705357</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>请根据上下四分位数分割，将体重分为high、normal、low三组，统计身高的均值。</p>
</div></blockquote>
</div>
<p>从索引可以看出，其实最后产生的结果就是按照条件列表中元素的值（此处是 <code class="docutils literal notranslate"><span class="pre">True</span></code> 和 <code class="docutils literal notranslate"><span class="pre">False</span></code> ）来分组，下面用随机传入字母序列来验证这一想法：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [8]: </span><span class="n">item</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="s1">&#39;abc&#39;</span><span class="p">),</span> <span class="n">df</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>

<span class="gp">In [9]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="n">item</span><span class="p">)[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gh">Out[9]: </span>
<span class="go">a    163.924242</span>
<span class="go">b    162.928814</span>
<span class="go">c    162.708621</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
<p>此处的索引就是原先item中的元素，如果传入多个序列进入 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> ，那么最后分组的依据就是这两个序列对应行的唯一组合：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [10]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="n">condition</span><span class="p">,</span> <span class="n">item</span><span class="p">])[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gh">Out[10]: </span>
<span class="go">Weight   </span>
<span class="go">False   a    160.193617</span>
<span class="go">        b    158.921951</span>
<span class="go">        c    157.756410</span>
<span class="go">True    a    173.152632</span>
<span class="go">        b    172.055556</span>
<span class="go">        c    172.873684</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
<p>由此可以看出，之前传入列名只是一种简便的记号，事实上等价于传入的是一个或多个列，最后分组的依据来自于数据来源组合的unique值，通过 <code class="docutils literal notranslate"><span class="pre">drop_duplicates</span></code> 就能知道具体的组类别：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [11]: </span><span class="n">df</span><span class="p">[[</span><span class="s1">&#39;School&#39;</span><span class="p">,</span> <span class="s1">&#39;Gender&#39;</span><span class="p">]]</span><span class="o">.</span><span class="n">drop_duplicates</span><span class="p">()</span>
<span class="gh">Out[11]: </span>
<span class="go">                           School  Gender</span>
<span class="go">0   Shanghai Jiao Tong University  Female</span>
<span class="go">1               Peking University    Male</span>
<span class="go">2   Shanghai Jiao Tong University    Male</span>
<span class="go">3                Fudan University  Female</span>
<span class="go">4                Fudan University    Male</span>
<span class="go">5             Tsinghua University  Female</span>
<span class="go">9               Peking University  Female</span>
<span class="go">16            Tsinghua University    Male</span>

<span class="gp">In [12]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="n">df</span><span class="p">[</span><span class="s1">&#39;School&#39;</span><span class="p">],</span> <span class="n">df</span><span class="p">[</span><span class="s1">&#39;Gender&#39;</span><span class="p">]])[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gh">Out[12]: </span>
<span class="go">School                         Gender</span>
<span class="go">Fudan University               Female    158.776923</span>
<span class="go">                               Male      174.212500</span>
<span class="go">Peking University              Female    158.666667</span>
<span class="go">                               Male      172.030000</span>
<span class="go">Shanghai Jiao Tong University  Female    159.122500</span>
<span class="go">                               Male      176.760000</span>
<span class="go">Tsinghua University            Female    159.753333</span>
<span class="go">                               Male      171.638889</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
</section>
<section id="groupby">
<h3>3. Groupby对象<a class="headerlink" href="#groupby" title="Permalink to this heading">#</a></h3>
<p>能够注意到，最终具体做分组操作时，所调用的方法都来自于 <code class="docutils literal notranslate"><span class="pre">pandas</span></code> 中的 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象，这个对象上定义了许多方法，也具有一些方便的属性。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [13]: </span><span class="n">gb</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">&#39;School&#39;</span><span class="p">,</span> <span class="s1">&#39;Grade&#39;</span><span class="p">])</span>

<span class="gp">In [14]: </span><span class="n">gb</span>
<span class="gh">Out[14]: </span><span class="go">&lt;pandas.core.groupby.generic.DataFrameGroupBy object at 0x00000292283D31C0&gt;</span>
</pre></div>
</div>
<p>通过 <code class="docutils literal notranslate"><span class="pre">ngroups</span></code> 属性，可以得到分组个数：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [15]: </span><span class="n">gb</span><span class="o">.</span><span class="n">ngroups</span>
<span class="gh">Out[15]: </span><span class="go">16</span>
</pre></div>
</div>
<p>通过 <code class="docutils literal notranslate"><span class="pre">groups</span></code> 属性，可以返回从 <span class="red">组名</span> 映射到 <span class="red">组索引列表</span> 的字典：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [16]: </span><span class="n">res</span> <span class="o">=</span> <span class="n">gb</span><span class="o">.</span><span class="n">groups</span>

<span class="gp">In [17]: </span><span class="n">res</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span> <span class="c1"># 字典的值由于是索引，元素个数过多，此处只展示字典的键</span>
<span class="gh">Out[17]: </span><span class="go">dict_keys([(&#39;Fudan University&#39;, &#39;Freshman&#39;), (&#39;Fudan University&#39;, &#39;Junior&#39;), (&#39;Fudan University&#39;, &#39;Senior&#39;), (&#39;Fudan University&#39;, &#39;Sophomore&#39;), (&#39;Peking University&#39;, &#39;Freshman&#39;), (&#39;Peking University&#39;, &#39;Junior&#39;), (&#39;Peking University&#39;, &#39;Senior&#39;), (&#39;Peking University&#39;, &#39;Sophomore&#39;), (&#39;Shanghai Jiao Tong University&#39;, &#39;Freshman&#39;), (&#39;Shanghai Jiao Tong University&#39;, &#39;Junior&#39;), (&#39;Shanghai Jiao Tong University&#39;, &#39;Senior&#39;), (&#39;Shanghai Jiao Tong University&#39;, &#39;Sophomore&#39;), (&#39;Tsinghua University&#39;, &#39;Freshman&#39;), (&#39;Tsinghua University&#39;, &#39;Junior&#39;), (&#39;Tsinghua University&#39;, &#39;Senior&#39;), (&#39;Tsinghua University&#39;, &#39;Sophomore&#39;)])</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>上一小节介绍了可以通过 <code class="docutils literal notranslate"><span class="pre">drop_duplicates</span></code> 得到具体的组类别，现请用 <code class="docutils literal notranslate"><span class="pre">groups</span></code> 属性完成类似的功能。</p>
</div></blockquote>
</div>
<p>当 <code class="docutils literal notranslate"><span class="pre">size</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 的属性时，返回的是表长乘以表宽的大小，但在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象上表示统计每个组的元素个数：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [18]: </span><span class="n">gb</span><span class="o">.</span><span class="n">size</span><span class="p">()</span>
<span class="gh">Out[18]: </span>
<span class="go">School                         Grade    </span>
<span class="go">Fudan University               Freshman      9</span>
<span class="go">                               Junior       12</span>
<span class="go">                               Senior       11</span>
<span class="go">                               Sophomore     8</span>
<span class="go">Peking University              Freshman     13</span>
<span class="go">                               Junior        8</span>
<span class="go">                               Senior        8</span>
<span class="go">                               Sophomore     5</span>
<span class="go">Shanghai Jiao Tong University  Freshman     13</span>
<span class="go">                               Junior       17</span>
<span class="go">                               Senior       22</span>
<span class="go">                               Sophomore     5</span>
<span class="go">Tsinghua University            Freshman     17</span>
<span class="go">                               Junior       22</span>
<span class="go">                               Senior       14</span>
<span class="go">                               Sophomore    16</span>
<span class="go">dtype: int64</span>
</pre></div>
</div>
<p>通过 <code class="docutils literal notranslate"><span class="pre">get_group</span></code> 方法可以直接获取所在组对应的行，此时必须知道组的具体名字：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [19]: </span><span class="n">gb</span><span class="o">.</span><span class="n">get_group</span><span class="p">((</span><span class="s1">&#39;Fudan University&#39;</span><span class="p">,</span> <span class="s1">&#39;Freshman&#39;</span><span class="p">))</span><span class="o">.</span><span class="n">iloc</span><span class="p">[:</span><span class="mi">3</span><span class="p">,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span> <span class="c1"># 展示一部分</span>
<span class="gh">Out[19]: </span>
<span class="go">              School     Grade             Name</span>
<span class="go">15  Fudan University  Freshman  Changqiang Yang</span>
<span class="go">28  Fudan University  Freshman     Gaoqiang Qin</span>
<span class="go">63  Fudan University  Freshman     Gaofeng Zhao</span>
</pre></div>
</div>
<p>这里列出了2个属性和2个方法，而先前的 <code class="docutils literal notranslate"><span class="pre">mean</span></code> 、 <code class="docutils literal notranslate"><span class="pre">median</span></code> 都是 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象上的方法，这些函数和许多其他函数的操作具有高度相似性，将在之后的小节进行专门介绍。</p>
</section>
<section id="id5">
<h3>4. 分组的三大操作<a class="headerlink" href="#id5" title="Permalink to this heading">#</a></h3>
<p>熟悉了一些分组的基本知识后，重新回到开头举的三个例子，可能会发现一些端倪，即这三种类型分组返回的数据型态并不一样：</p>
<ul class="simple">
<li><p>第一个例子中，每一个组返回一个标量值，可以是平均值、中位数、组容量 <code class="docutils literal notranslate"><span class="pre">size</span></code> 等</p></li>
<li><p>第二个例子中，做了原序列的标准化处理，也就是说每组返回的是一个 <code class="docutils literal notranslate"><span class="pre">Series</span></code> 类型</p></li>
<li><p>第三个例子中，既不是标量也不是序列，返回的整个组所在行的本身，即返回了 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 类型</p></li>
</ul>
<p>由此，引申出分组的三大操作：聚合、变换和过滤，分别对应了三个例子的操作，下面就要分别介绍相应的 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 、 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 和 <code class="docutils literal notranslate"><span class="pre">filter</span></code> 函数及其操作。</p>
</section>
</section>
<section id="id6">
<h2>二、聚合函数<a class="headerlink" href="#id6" title="Permalink to this heading">#</a></h2>
<section id="id7">
<h3>1. 内置聚合函数<a class="headerlink" href="#id7" title="Permalink to this heading">#</a></h3>
<p>在介绍agg之前，首先要了解一些直接定义在groupby对象的聚合函数，因为它的速度基本都会经过内部的优化，使用功能时应当优先考虑。根据返回标量值的原则，包括如下函数： <code class="docutils literal notranslate"><span class="pre">max/min/mean/median/count/all/any/idxmax/idxmin/mad/nunique/skew/quantile/sum/std/var/sem/size/prod</span></code> 。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [20]: </span><span class="n">gb</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s1">&#39;Gender&#39;</span><span class="p">)[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span>

<span class="gp">In [21]: </span><span class="n">gb</span><span class="o">.</span><span class="n">idxmin</span><span class="p">()</span>
<span class="gh">Out[21]: </span>
<span class="go">Gender</span>
<span class="go">Female    143</span>
<span class="go">Male      199</span>
<span class="go">Name: Height, dtype: int64</span>

<span class="gp">In [22]: </span><span class="n">gb</span><span class="o">.</span><span class="n">quantile</span><span class="p">(</span><span class="mf">0.95</span><span class="p">)</span>
<span class="gh">Out[22]: </span>
<span class="go">Gender</span>
<span class="go">Female    166.8</span>
<span class="go">Male      185.9</span>
<span class="go">Name: Height, dtype: float64</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>请查阅文档，明确 <code class="docutils literal notranslate"><span class="pre">all/any/mad/skew/sem/prod</span></code> 函数的含义。</p>
</div></blockquote>
</div>
<p>这些聚合函数当传入的数据来源包含多个列时，将按照列进行迭代计算：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [23]: </span><span class="n">gb</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s1">&#39;Gender&#39;</span><span class="p">)[[</span><span class="s1">&#39;Height&#39;</span><span class="p">,</span> <span class="s1">&#39;Weight&#39;</span><span class="p">]]</span>

<span class="gp">In [24]: </span><span class="n">gb</span><span class="o">.</span><span class="n">max</span><span class="p">()</span>
<span class="gh">Out[24]: </span>
<span class="go">        Height  Weight</span>
<span class="go">Gender                </span>
<span class="go">Female   170.2    63.0</span>
<span class="go">Male     193.9    89.0</span>
</pre></div>
</div>
</section>
<section id="agg">
<h3>2. agg方法<a class="headerlink" href="#agg" title="Permalink to this heading">#</a></h3>
<p>虽然在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象上定义了许多方便的函数，但仍然有以下不便之处：</p>
<ul class="simple">
<li><p>无法同时使用多个函数</p></li>
<li><p>无法对特定的列使用特定的聚合函数</p></li>
<li><p>无法使用自定义的聚合函数</p></li>
<li><p>无法直接对结果的列名在聚合前进行自定义命名</p></li>
</ul>
<p>下面说明如何通过 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 函数解决这四类问题：</p>
<p>【a】使用多个函数</p>
<p>当使用多个聚合函数时，需要用列表的形式把内置聚合函数对应的字符串传入，先前提到的所有字符串都是合法的。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [25]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">([</span><span class="s1">&#39;sum&#39;</span><span class="p">,</span> <span class="s1">&#39;idxmax&#39;</span><span class="p">,</span> <span class="s1">&#39;skew&#39;</span><span class="p">])</span>
<span class="gh">Out[25]: </span>
<span class="go">         Height                   Weight                 </span>
<span class="go">            sum idxmax      skew     sum idxmax      skew</span>
<span class="go">Gender                                                   </span>
<span class="go">Female  21014.0     28 -0.219253  6469.0     28 -0.268482</span>
<span class="go">Male     8854.9    193  0.437535  3929.0      2 -0.332393</span>
</pre></div>
</div>
<p>从结果看，此时的列索引为多级索引，第一层为数据源，第二层为使用的聚合方法，分别逐一对列使用聚合，因此结果为6列。</p>
<p>【b】对特定的列使用特定的聚合函数</p>
<p>对于方法和列的特殊对应，可以通过构造字典传入 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 中实现，其中字典以列名为键，以聚合字符串或字符串列表为值。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [26]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">({</span><span class="s1">&#39;Height&#39;</span><span class="p">:[</span><span class="s1">&#39;mean&#39;</span><span class="p">,</span><span class="s1">&#39;max&#39;</span><span class="p">],</span> <span class="s1">&#39;Weight&#39;</span><span class="p">:</span><span class="s1">&#39;count&#39;</span><span class="p">})</span>
<span class="gh">Out[26]: </span>
<span class="go">           Height        Weight</span>
<span class="go">             mean    max  count</span>
<span class="go">Gender                         </span>
<span class="go">Female  159.19697  170.2    135</span>
<span class="go">Male    173.62549  193.9     54</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>请使用【b】中的传入字典的方法完成【a】中等价的聚合任务。</p>
</div></blockquote>
</div>
<p>【c】使用自定义函数</p>
<p>在 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 中可以使用具体的自定义函数， <span class="red">需要注意传入函数的参数是之前数据源中的列，逐列进行计算</span> 。下面分组计算身高和体重的极差：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [27]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">max</span><span class="p">()</span><span class="o">-</span><span class="n">x</span><span class="o">.</span><span class="n">min</span><span class="p">())</span>
<span class="gh">Out[27]: </span>
<span class="go">        Height  Weight</span>
<span class="go">Gender                </span>
<span class="go">Female    24.8    29.0</span>
<span class="go">Male      38.2    38.0</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象中可以使用 <code class="docutils literal notranslate"><span class="pre">describe</span></code> 方法进行统计信息汇总，请同时使用多个聚合函数，完成与该方法相同的功能。</p>
</div></blockquote>
</div>
<p>由于传入的是序列，因此序列上的方法和属性都是可以在函数中使用的，只需保证返回值是标量即可。下面的例子是指，如果组的指标均值，超过该指标的总体均值，返回High，否则返回Low。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [28]: </span><span class="k">def</span> <span class="nf">my_func</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="gp">   ....: </span>    <span class="n">res</span> <span class="o">=</span> <span class="s1">&#39;High&#39;</span>
<span class="gp">   ....: </span>    <span class="k">if</span> <span class="n">s</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span> <span class="o">&lt;=</span> <span class="n">df</span><span class="p">[</span><span class="n">s</span><span class="o">.</span><span class="n">name</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">():</span>
<span class="gp">   ....: </span>        <span class="n">res</span> <span class="o">=</span> <span class="s1">&#39;Low&#39;</span>
<span class="gp">   ....: </span>    <span class="k">return</span> <span class="n">res</span>
<span class="gp">   ....: </span>

<span class="gp">In [29]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">(</span><span class="n">my_func</span><span class="p">)</span>
<span class="gh">Out[29]: </span>
<span class="go">       Height Weight</span>
<span class="go">Gender              </span>
<span class="go">Female    Low    Low</span>
<span class="go">Male     High   High</span>
</pre></div>
</div>
<p>【d】聚合结果重命名</p>
<p>如果想要对聚合结果的列名进行重命名，只需要将上述函数的位置改写成元组，元组的第一个元素为新的名字，第二个位置为原来的函数，包括聚合字符串和自定义函数，现举若干例子说明：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [30]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">([(</span><span class="s1">&#39;range&#39;</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">max</span><span class="p">()</span><span class="o">-</span><span class="n">x</span><span class="o">.</span><span class="n">min</span><span class="p">()),</span> <span class="p">(</span><span class="s1">&#39;my_sum&#39;</span><span class="p">,</span> <span class="s1">&#39;sum&#39;</span><span class="p">)])</span>
<span class="gh">Out[30]: </span>
<span class="go">       Height          Weight        </span>
<span class="go">        range   my_sum  range  my_sum</span>
<span class="go">Gender                               </span>
<span class="go">Female   24.8  21014.0   29.0  6469.0</span>
<span class="go">Male     38.2   8854.9   38.0  3929.0</span>
</pre></div>
</div>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [31]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">({</span><span class="s1">&#39;Height&#39;</span><span class="p">:</span> <span class="p">[(</span><span class="s1">&#39;my_func&#39;</span><span class="p">,</span> <span class="n">my_func</span><span class="p">),</span> <span class="s1">&#39;sum&#39;</span><span class="p">],</span>
<span class="gp">   ....: </span>        <span class="s1">&#39;Weight&#39;</span><span class="p">:</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span><span class="n">x</span><span class="o">.</span><span class="n">max</span><span class="p">()})</span>
<span class="gp">   ....: </span>
<span class="gh">Out[31]: </span>
<span class="go">        Height            Weight</span>
<span class="go">       my_func      sum &lt;lambda&gt;</span>
<span class="go">Gender                          </span>
<span class="go">Female     Low  21014.0     63.0</span>
<span class="go">Male      High   8854.9     89.0</span>
</pre></div>
</div>
<p>另外需要注意，使用对一个或者多个列使用单个聚合的时候，重命名需要加方括号，否则就不知道是新的名字还是手误输错的内置函数字符串：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [32]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">([(</span><span class="s1">&#39;my_sum&#39;</span><span class="p">,</span> <span class="s1">&#39;sum&#39;</span><span class="p">)])</span>
<span class="gh">Out[32]: </span>
<span class="go">         Height  Weight</span>
<span class="go">         my_sum  my_sum</span>
<span class="go">Gender                 </span>
<span class="go">Female  21014.0  6469.0</span>
<span class="go">Male     8854.9  3929.0</span>
</pre></div>
</div>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [33]: </span><span class="n">gb</span><span class="o">.</span><span class="n">agg</span><span class="p">({</span><span class="s1">&#39;Height&#39;</span><span class="p">:</span> <span class="p">[(</span><span class="s1">&#39;my_func&#39;</span><span class="p">,</span> <span class="n">my_func</span><span class="p">),</span> <span class="s1">&#39;sum&#39;</span><span class="p">],</span>
<span class="gp">   ....: </span>        <span class="s1">&#39;Weight&#39;</span><span class="p">:</span> <span class="p">[(</span><span class="s1">&#39;range&#39;</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span><span class="n">x</span><span class="o">.</span><span class="n">max</span><span class="p">())]})</span>
<span class="gp">   ....: </span>
<span class="gh">Out[33]: </span>
<span class="go">        Height          Weight</span>
<span class="go">       my_func      sum  range</span>
<span class="go">Gender                        </span>
<span class="go">Female     Low  21014.0   63.0</span>
<span class="go">Male      High   8854.9   89.0</span>
</pre></div>
</div>
</section>
</section>
<section id="id8">
<h2>三、变换和过滤<a class="headerlink" href="#id8" title="Permalink to this heading">#</a></h2>
<section id="transform">
<h3>1. 变换函数与transform方法<a class="headerlink" href="#transform" title="Permalink to this heading">#</a></h3>
<p>变换函数的返回值为同长度的序列，最常用的内置变换函数是累计函数： <code class="docutils literal notranslate"><span class="pre">cumcount/cumsum/cumprod/cummax/cummin</span></code> ，它们的使用方式和聚合函数类似，只不过完成的是组内累计操作。此外在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象上还定义了填充类和滑窗类的变换函数，这些函数的一般形式将会分别在第七章和第十章中讨论，此处略过。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [34]: </span><span class="n">gb</span><span class="o">.</span><span class="n">cummax</span><span class="p">()</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gh">Out[34]: </span>
<span class="go">   Height  Weight</span>
<span class="go">0   158.9    46.0</span>
<span class="go">1   166.5    70.0</span>
<span class="go">2   188.9    89.0</span>
<span class="go">3     NaN    46.0</span>
<span class="go">4   188.9    89.0</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象中， <code class="docutils literal notranslate"><span class="pre">rank</span></code> 方法也是一个实用的变换函数，请查阅它的功能并给出一个使用的例子。</p>
</div></blockquote>
</div>
<p>当用自定义变换时需要使用 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 方法，被调用的自定义函数， <span class="red">其传入值为数据源的序列</span> ，与 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 的传入类型是一致的，其最后的返回结果是行列索引与数据源一致的 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 。</p>
<p>现对身高和体重进行分组标准化，即减去组均值后除以组的标准差：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [35]: </span><span class="n">gb</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="n">x</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span><span class="o">/</span><span class="n">x</span><span class="o">.</span><span class="n">std</span><span class="p">())</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gh">Out[35]: </span>
<span class="go">     Height    Weight</span>
<span class="go">0 -0.058760 -0.354888</span>
<span class="go">1 -1.010925 -0.355000</span>
<span class="go">2  2.167063  2.089498</span>
<span class="go">3       NaN -1.279789</span>
<span class="go">4  0.053133  0.159631</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>对于 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 方法无法像 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 一样，通过传入字典来对指定列使用特定的变换，如果需要在一次 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 的调用中实现这种功能，请给出解决方案。</p>
</div></blockquote>
</div>
<p>前面提到了 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 只能返回同长度的序列，但事实上还可以返回一个标量，这会使得结果被广播到其所在的整个组，这种 <span class="red">标量广播</span> 的技巧在特征工程中是非常常见的。例如，构造两列新特征来分别表示样本所在性别组的身高均值和体重均值：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [36]: </span><span class="n">gb</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="s1">&#39;mean&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span> <span class="c1"># 传入返回标量的函数也是可以的</span>
<span class="gh">Out[36]: </span>
<span class="go">      Height     Weight</span>
<span class="go">0  159.19697  47.918519</span>
<span class="go">1  173.62549  72.759259</span>
<span class="go">2  173.62549  72.759259</span>
<span class="go">3  159.19697  47.918519</span>
<span class="go">4  173.62549  72.759259</span>
</pre></div>
</div>
</section>
<section id="id9">
<h3>2. 组索引与过滤<a class="headerlink" href="#id9" title="Permalink to this heading">#</a></h3>
<p>在上一章中介绍了索引的用法，那么索引和过滤有什么区别呢？</p>
<p>过滤在分组中是对于组的过滤，而索引是对于行的过滤，在第二章中的返回值，无论是布尔列表还是元素列表或者位置列表，本质上都是对于行的筛选，即如果符合筛选条件的则选入结果表，否则不选入。</p>
<p>组过滤作为行过滤的推广，指的是如果对一个组的全体所在行进行统计的结果返回 <code class="docutils literal notranslate"><span class="pre">True</span></code> 则会被保留， <code class="docutils literal notranslate"><span class="pre">False</span></code> 则该组会被过滤，最后把所有未被过滤的组其对应的所在行拼接起来作为 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 返回。</p>
<p>在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象中，定义了 <code class="docutils literal notranslate"><span class="pre">filter</span></code> 方法进行组的筛选，其中自定义函数的输入参数为数据源构成的 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 本身，在之前例子中定义的 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象中，传入的就是 <code class="docutils literal notranslate"><span class="pre">df[['Height',</span> <span class="pre">'Weight']]</span></code> ，因此所有表方法和属性都可以在自定义函数中相应地使用，同时只需保证自定义函数的返回为布尔值即可。</p>
<p>例如，在原表中通过过滤得到所有容量大于100的组：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [37]: </span><span class="n">gb</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&gt;</span> <span class="mi">100</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gh">Out[37]: </span>
<span class="go">   Height  Weight</span>
<span class="go">0   158.9    46.0</span>
<span class="go">3     NaN    41.0</span>
<span class="go">5   158.0    51.0</span>
<span class="go">6   162.5    52.0</span>
<span class="go">7   161.9    50.0</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>从概念上说，索引功能是组过滤功能的子集，请使用 <code class="docutils literal notranslate"><span class="pre">filter</span></code> 函数完成 <code class="docutils literal notranslate"><span class="pre">loc[.]</span></code> 的功能，这里假设 ” <code class="docutils literal notranslate"><span class="pre">.</span></code> “是元素列表。</p>
</div></blockquote>
</div>
</section>
</section>
<section id="id10">
<h2>四、跨列分组<a class="headerlink" href="#id10" title="Permalink to this heading">#</a></h2>
<section id="apply">
<h3>1. apply的引入<a class="headerlink" href="#apply" title="Permalink to this heading">#</a></h3>
<p>之前几节介绍了三大分组操作，但事实上还有一种常见的分组场景，无法用前面介绍的任何一种方法处理，例如现在如下定义身体质量指数BMI：</p>
<div class="math notranslate nohighlight">
\[{\rm BMI} = {\rm\frac{Weight}{Height^2}}\]</div>
<p>其中体重和身高的单位分别为千克和米，需要分组计算组BMI的均值。</p>
<p>首先，这显然不是过滤操作，因此 <code class="docutils literal notranslate"><span class="pre">filter</span></code> 不符合要求；其次，返回的均值是标量而不是序列，因此 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 不符合要求；最后，似乎使用 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 函数能够处理，但是之前强调过聚合函数是逐列处理的，而不能够 <span class="red">多列数据同时处理</span> 。由此，引出了 <code class="docutils literal notranslate"><span class="pre">apply</span></code> 函数来解决这一问题。</p>
</section>
<section id="id11">
<h3>2. apply的使用<a class="headerlink" href="#id11" title="Permalink to this heading">#</a></h3>
<p>在设计上， <code class="docutils literal notranslate"><span class="pre">apply</span></code> 的自定义函数传入参数与 <code class="docutils literal notranslate"><span class="pre">filter</span></code> 完全一致，只不过后者只允许返回布尔值。现如下解决上述计算问题：</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [38]: </span><span class="k">def</span> <span class="nf">BMI</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="gp">   ....: </span>    <span class="n">Height</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;Height&#39;</span><span class="p">]</span><span class="o">/</span><span class="mi">100</span>
<span class="gp">   ....: </span>    <span class="n">Weight</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;Weight&#39;</span><span class="p">]</span>
<span class="gp">   ....: </span>    <span class="n">BMI_value</span> <span class="o">=</span> <span class="n">Weight</span><span class="o">/</span><span class="n">Height</span><span class="o">**</span><span class="mi">2</span>
<span class="gp">   ....: </span>    <span class="k">return</span> <span class="n">BMI_value</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gp">   ....: </span>

<span class="gp">In [39]: </span><span class="n">gb</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="n">BMI</span><span class="p">)</span>
<span class="gh">Out[39]: </span>
<span class="go">Gender</span>
<span class="go">Female    18.860930</span>
<span class="go">Male      24.318654</span>
<span class="go">dtype: float64</span>
</pre></div>
</div>
<p>除了返回标量之外， <code class="docutils literal notranslate"><span class="pre">apply</span></code> 方法还可以返回一维 <code class="docutils literal notranslate"><span class="pre">Series</span></code> 和二维 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> ，但它们产生的数据框维数和多级索引的层数应当如何变化？下面举三组例子就非常容易明白结果是如何生成的：</p>
<p>【a】标量情况：结果得到的是 <code class="docutils literal notranslate"><span class="pre">Series</span></code> ，索引与 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 的结果一致</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [40]: </span><span class="n">gb</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">&#39;Gender&#39;</span><span class="p">,</span><span class="s1">&#39;Test_Number&#39;</span><span class="p">])[[</span><span class="s1">&#39;Height&#39;</span><span class="p">,</span><span class="s1">&#39;Weight&#39;</span><span class="p">]]</span>

<span class="gp">In [41]: </span><span class="n">gb</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="mi">0</span><span class="p">)</span>
<span class="gh">Out[41]: </span>
<span class="go">Gender  Test_Number</span>
<span class="go">Female  1              0</span>
<span class="go">        2              0</span>
<span class="go">        3              0</span>
<span class="go">Male    1              0</span>
<span class="go">        2              0</span>
<span class="go">        3              0</span>
<span class="go">dtype: int64</span>

<span class="gp">In [42]: </span><span class="n">gb</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">])</span> <span class="c1"># 虽然是列表，但是作为返回值仍然看作标量</span>
<span class="gh">Out[42]: </span>
<span class="go">Gender  Test_Number</span>
<span class="go">Female  1              [0, 0]</span>
<span class="go">        2              [0, 0]</span>
<span class="go">        3              [0, 0]</span>
<span class="go">Male    1              [0, 0]</span>
<span class="go">        2              [0, 0]</span>
<span class="go">        3              [0, 0]</span>
<span class="go">dtype: object</span>
</pre></div>
</div>
<p>【b】 <code class="docutils literal notranslate"><span class="pre">Series</span></code> 情况：得到的是 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> ，行索引与标量情况一致，列索引为 <code class="docutils literal notranslate"><span class="pre">Series</span></code> 的索引</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [43]: </span><span class="n">gb</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span><span class="n">index</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;a&#39;</span><span class="p">,</span><span class="s1">&#39;b&#39;</span><span class="p">]))</span>
<span class="gh">Out[43]: </span>
<span class="go">                    a  b</span>
<span class="go">Gender Test_Number      </span>
<span class="go">Female 1            0  0</span>
<span class="go">       2            0  0</span>
<span class="go">       3            0  0</span>
<span class="go">Male   1            0  0</span>
<span class="go">       2            0  0</span>
<span class="go">       3            0  0</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>请尝试在 <code class="docutils literal notranslate"><span class="pre">apply</span></code> 传入的自定义函数中，根据组的某些特征返回相同长度但索引不同的 <code class="docutils literal notranslate"><span class="pre">Series</span></code> ，会报错吗？</p>
</div></blockquote>
</div>
<p>【c】 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 情况：得到的是 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> ，行索引最内层在每个组原先 <code class="docutils literal notranslate"><span class="pre">agg</span></code> 的结果索引上，再加一层返回的 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 行索引，同时分组结果 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 的列索引和返回的 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> 列索引一致。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [44]: </span><span class="n">gb</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">)),</span>
<span class="gp">   ....: </span>                                <span class="n">index</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;a&#39;</span><span class="p">,</span><span class="s1">&#39;b&#39;</span><span class="p">],</span>
<span class="gp">   ....: </span>                                <span class="n">columns</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">Index</span><span class="p">([(</span><span class="s1">&#39;w&#39;</span><span class="p">,</span><span class="s1">&#39;x&#39;</span><span class="p">),(</span><span class="s1">&#39;y&#39;</span><span class="p">,</span><span class="s1">&#39;z&#39;</span><span class="p">)])))</span>
<span class="gp">   ....: </span>
<span class="gh">Out[44]: </span>
<span class="go">                        w    y</span>
<span class="go">                        x    z</span>
<span class="go">Gender Test_Number            </span>
<span class="go">Female 1           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
<span class="go">       2           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
<span class="go">       3           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
<span class="go">Male   1           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
<span class="go">       2           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
<span class="go">       3           a  1.0  1.0</span>
<span class="go">                   b  1.0  1.0</span>
</pre></div>
</div>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>请尝试在 <code class="docutils literal notranslate"><span class="pre">apply</span></code> 传入的自定义函数中，根据组的某些特征返回相同大小但列索引不同的 <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> ，会报错吗？如果只是行索引不同，会报错吗？</p>
</div></blockquote>
</div>
<p>最后需要强调的是， <code class="docutils literal notranslate"><span class="pre">apply</span></code> 函数的灵活性是以牺牲一定性能为代价换得的，除非需要使用跨列处理的分组处理，否则应当使用其他专门设计的 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象方法，否则在性能上会存在较大的差距。同时，在使用聚合函数和变换函数时，也应当优先使用内置函数，它们经过了高度的性能优化，一般而言在速度上都会快于用自定义函数来实现。</p>
<div class="hint admonition">
<p class="admonition-title">练一练</p>
<blockquote>
<div><p>在 <code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象中还定义了 <code class="docutils literal notranslate"><span class="pre">cov</span></code> 和 <code class="docutils literal notranslate"><span class="pre">corr</span></code> 函数，从概念上说也属于跨列的分组处理。请利用之前定义的 <code class="docutils literal notranslate"><span class="pre">gb</span></code> 对象，使用apply函数实现与 <code class="docutils literal notranslate"><span class="pre">gb.cov()</span></code> 同样的功能并比较它们的性能。</p>
</div></blockquote>
</div>
</section>
</section>
<section id="id12">
<h2>五、练习<a class="headerlink" href="#id12" title="Permalink to this heading">#</a></h2>
<section id="ex1">
<h3>Ex1：汽车数据集<a class="headerlink" href="#ex1" title="Permalink to this heading">#</a></h3>
<p>现有一份汽车数据集，其中 <code class="docutils literal notranslate"><span class="pre">Brand,</span> <span class="pre">Disp.,</span> <span class="pre">HP</span></code> 分别代表汽车品牌、发动机蓄量、发动机输出。</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [45]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s1">&#39;data/car.csv&#39;</span><span class="p">)</span>

<span class="gp">In [46]: </span><span class="n">df</span><span class="o">.</span><span class="n">head</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="gh">Out[46]: </span>
<span class="go">             Brand  Price Country  Reliability  Mileage   Type  Weight  Disp.   HP</span>
<span class="go">0   Eagle Summit 4   8895     USA          4.0       33  Small    2560     97  113</span>
<span class="go">1  Ford Escort   4   7402     USA          2.0       33  Small    2345    114   90</span>
<span class="go">2   Ford Festiva 4   6319   Korea          4.0       37  Small    1845     81   63</span>
</pre></div>
</div>
<ol class="arabic simple">
<li><p>先过滤出所属 <code class="docutils literal notranslate"><span class="pre">Country</span></code> 数超过2个的汽车，即若该汽车的 <code class="docutils literal notranslate"><span class="pre">Country</span></code> 在总体数据集中出现次数不超过2则剔除，再按 <code class="docutils literal notranslate"><span class="pre">Country</span></code> 分组计算价格均值、价格变异系数、该 <code class="docutils literal notranslate"><span class="pre">Country</span></code> 的汽车数量，其中变异系数的计算方法是标准差除以均值，并在结果中把变异系数重命名为 <code class="docutils literal notranslate"><span class="pre">CoV</span></code> 。</p></li>
<li><p>按照表中位置的前三分之一、中间三分之一和后三分之一分组，统计 <code class="docutils literal notranslate"><span class="pre">Price</span></code> 的均值。</p></li>
<li><p>对类型 <code class="docutils literal notranslate"><span class="pre">Type</span></code> 分组，对 <code class="docutils literal notranslate"><span class="pre">Price</span></code> 和 <code class="docutils literal notranslate"><span class="pre">HP</span></code> 分别计算最大值和最小值，结果会产生多级索引，请用下划线把多级列索引合并为单层索引。</p></li>
<li><p>对类型 <code class="docutils literal notranslate"><span class="pre">Type</span></code> 分组，对 <code class="docutils literal notranslate"><span class="pre">HP</span></code> 进行组内的 <code class="docutils literal notranslate"><span class="pre">min-max</span></code> 归一化。</p></li>
<li><p>对类型 <code class="docutils literal notranslate"><span class="pre">Type</span></code> 分组，计算 <code class="docutils literal notranslate"><span class="pre">Disp.</span></code> 与 <code class="docutils literal notranslate"><span class="pre">HP</span></code> 的相关系数。</p></li>
</ol>
</section>
<section id="ex2-transform">
<h3>Ex2：实现transform函数<a class="headerlink" href="#ex2-transform" title="Permalink to this heading">#</a></h3>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">groupby</span></code> 对象的构造方法是 <code class="docutils literal notranslate"><span class="pre">my_groupby(df,</span> <span class="pre">group_cols)</span></code></p></li>
<li><p>支持单列分组与多列分组</p></li>
<li><p>支持带有标量广播的 <code class="docutils literal notranslate"><span class="pre">my_groupby(df)[col].transform(my_func)</span></code> 功能</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pandas</span></code> 的 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 不能跨列计算，请支持此功能，即仍返回 <code class="docutils literal notranslate"><span class="pre">Series</span></code> 但 <code class="docutils literal notranslate"><span class="pre">col</span></code> 参数为多列</p></li>
<li><p>无需考虑性能与异常处理，只需实现上述功能，在给出测试样例的同时与 <code class="docutils literal notranslate"><span class="pre">pandas</span></code> 中的 <code class="docutils literal notranslate"><span class="pre">transform</span></code> 对比结果是否一致</p></li>
</ul>
</section>
</section>
</section>


              </article>
              

              
          </div>
          
      </div>
    </div>

  
  
  <!-- Scripts loaded after <body> so the DOM is not blocked -->
  <script src="../_static/scripts/pydata-sphinx-theme.js?digest=92025949c220c2e29695"></script>

<footer class="bd-footer"><div class="bd-footer__inner container">
  
  <div class="footer-item">
    <p class="copyright">
    &copy; Copyright 2020-2022, Datawhale, 耿远昊.<br>
</p>
  </div>
  
  <div class="footer-item">
    <p class="sphinx-version">
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.0.2.<br>
</p>
  </div>
  
</div>
</footer>
  </body>
</html>